Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support for sparse computations and sparse neural network layers with (custom) Caffe #142

Merged
merged 9 commits into from
Jul 4, 2016
Merged

Support for sparse computations and sparse neural network layers with (custom) Caffe #142

merged 9 commits into from
Jul 4, 2016

Conversation

beniz
Copy link
Collaborator

@beniz beniz commented Jun 6, 2016 •
edited
Loading

Neural embeddings have proved useful for a variety of tasks, such as text classification. However, sparse representations such as one-hot vectors or bag of words (BOW) still yield better results in practice on a series of datasets. This has been carefully measured in practice by one of our partners and ourselves as well.

This PR brings support for sparse representation and computation, for CPU and GPU, to be used with the text input connectors, and other forthcoming connectors (see #112).

Sparse computations are realized by a custom version of Caffe, available from https://github.com/beniz/caffe/tree/master_dd_integ_sparse, more details in #8.

Current status:

  • Sparse computations on CPU
  • Sparse computations on GPU
  • Sparse DataLayer (LMDB / LevelDB)
  • Sparse MemoryDataLayer
  • Sparse InnerProduct layer
  • Support for text connector on BOW models
  • Generic libSVM format support for sparse data
  • No multi-target regression with sparse inputs since the matrix cannot be sliced at the moment
  • Support for sparse logistic regression
  • Unit tests

Example:
Use sparse:true parameter to input connector.

This PR is sponsored by ioSquare.

@beniz beniz self-assigned this Jun 6, 2016
@beniz beniz changed the title Support for sparse computations and neural network layers with (custom) Caffe Support for sparse computations and sparse neural network layers with (custom) Caffe Jun 6, 2016
@beniz beniz mentioned this pull request Jun 6, 2016
Closed
@beniz beniz mentioned this pull request Jun 16, 2016
Open
@beniz
Copy link
Collaborator Author

beniz commented Jun 22, 2016 •
edited
Loading

Example using libsvm input file format for training and testing datasets with sparse computations:

  • setting up a service (notice the connector:svm):
curl -X PUT "http://localhost:8080/services/testsvm" -d "{\"mllib\":\"caffe\",\"description\":\"classification service\",\"type\":\"supervised\",\"parameters\":{\"input\":{\"connector\":\"svm\"},\"mllib\":{\"gpu\":true,\"template\":\"mlp\",\"layers\":[150,150,100],\"activation\":\"elu\",\"nclasses\":2,\"db\":true,\"dropout\":0.2}},\"model\":{\"templates\":\"../templates/caffe/\",\"repository\":\"/path/to/model\"}}"
  • training over a dataset in libsvm format:
curl -X POST "http://localhost:8080/train" -d "{\"service\":\"testsvm\",\"async\":true,\"parameters\":{\"mllib\":{\"gpu\":true,\"solver\":{\"test_initialization\":false,\"iterations\":5000,\"test_interval\":500,\"base_lr\":0.1,\"solver_type\":\"RMSPROP\"},\"net\":{\"batch_size\":512,\"test_batch_size\":32}},\"input\":{},\"output\":{\"measure\":[\"auc\",\"mcll\",\"f1\"]}},\"data\":[\"/path/to/train.svm\",\"/path/to/test.svm\"]}"

@beniz beniz merged commit f2e9acb into master Jul 4, 2016
@beniz beniz deleted the sparse_112 branch September 16, 2016 05:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@beniz beniz

Labels
datatype:text kind:API kind:C++ kind:connectors kind:neural net mllib:caffe other:sponsored type:enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@beniz